Regret Minimization in MDPs with Options without Prior Knowledge

Motivations I “Flat” RL : difficult to learn complex behaviours (eg, sequence of subgoals) ⇒ Humans abstract from low-level actions I Hierarchical RL : decompose large problems into smaller ones by imposing constraints on value function or policy I Possible implementation: options [Sutton et al., 1999] I Empirical observations: introducing options in an MDP can speed up learning but can also be harmful [Jong et al., 2008]. ⇒ Lack of theoretical motivation and understanding of options Theoretical analysis of learning with options I Adding options does not just reduce the space of stationary policies, the exploration is also greatly affected